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1.  Introduction. 


There  are  several  ways  of  developing  the  autoregressive  stochastic  proces  as  a  finite- 
parameter  model  for  time  series  analysis.  In  this  paper  we  obtain  the  properties  of  the  au¬ 
toregressive  process  from  a  stationary  stochastic  process  that  satisfies  the  simple  condition 
that  a  linear  combination  of  current  and  past  elements  of  the  process  is  independent  of  (or 
alternatively  uncorrelated  with)  all  earlier  elements  of  the  process.  This  approach  provides 
a  coherent,  clear,  and  rigorous  exposition  of  the  autoregressive  model.  The  stationarity  and 
independence  imply  that  the  roots  of  the  associated  polynomial  equation  are  less  than  1  in 
absolute  value.  The  existence  of  the  moving  average  representation  is  deduced  and  its  form  for 
distinct  roots.  The  Yule- Walker  equations,  which  are  derived,  determine  the  autocovariance 
sequence.  Another  set  of  parameters  consists  of  the  variance  of  the  process  and  the  partial 
autocorrelation  sequence. 

2.  Definition  of  the  Autoregressive  Process. 

A  stochastic  process  {yt},  which  consists  of  a  doubly  infinite  sequence  of  random  variables 
yt,  t  =  ...  ,—1,0,1,. . can  be  defined  by  means  of  the  distributions  of  all  finite  sets  of  these 

variables,  ytlt . . .,  yt„,  n  =  1, 2, _  The  stochastic  process  is  stationary  if  the  distribution  of 

yt1+k,  •  •  •  j  Vtn-\-h  is  the  same  as  the  distribution  of  yt1 , ... ,  yt„  for  h  =  . . . ,  — 1, 0, 1, . . .  and  n  = 
1,2,....  If  £ yf  <  oo,  then  £  yt  does  not  depend  on  t  and  we  write  £yj  =  n  and  £  (yt—fi)(yt+h—fA) 
does  not  depend  on  t  and  we  write 

(1)  £(yt-M)(y t+h  ~  »)  =  <r(h),  h  =...,- 1,0,1,.... 

Note  that  cr(  —  h )  =  cr(h).  The  quantity  c(/i)  is  known  as  the  autocovariance  of  order  h. 

We  now  consider  a  stationary  stochastic  process  {yt}  with  mean  such  that  for  some 
constants  /3q  =  l,  fix, . .  .,fip 


1 


(2) 


p 

i=0 

is  independent  of  yt-i,yt-2,  ■  ■  ■  for  t  =  . . . ,  —1, 0,1, _  Then  {u*}  is  a  stationary  stochastic 

process  because  the  distribution  of  utl+3, . . . ,  ufn+a  does  not  depend  on  s  (since  the  distribution 
of  yti+3,  yti+o-1,  •  •  • ,  Vh+3-p,  ■  ■  ■,  ytn+3,  •  •  • ,  ytn+3-p  does  not  depend  on  s).  Furthermore,  ut  is 
independent  of  ut_i  =  Y7j=o  Pifa-i-i  ~  A»),  «t-2  =  Y!j=oPj{Vt-j-2  ~  p),  •  •  ••  Thus  the  ut’s 
are  independently  and  identically  distributed.  Since  £yt  =  p,  then  £ut  =  0.  If  £yf  <  oo,  we 
can  write  £u^  =  <r2,  which  we  assume  to  be  positive.  (If  cr2  =  0,  the  process  satisfies  a  linear 
relation  with  probability  1  and  is  deterministic.)  The  process  {yt}  is  called  an  autoregressive 
process  of  order  p,  and  (2)  is  called  a  stochastic  difference  equation  of  degree  p. 

Now  multiply  the  left-hand  and  right-hand  sides  of  the  stochastic  difference  equation  (2) 
by  the  left-hand  and  right-hand  sides  of  yt  -  p  =  ut  -  0j(yt-j  -  p),  respectively,  and 

take  expected  values,  assuming  £ (yt  -  p)2  <  oo.  Since  ut  is  independent  of  yt-i, . . yt~P,  we 
obtain 


(3)  = 

i=o 

When  we  multiply  both  sides  of  (2)  by  yt~3  —  p  and  take  expected  values,  we  obtain 


p 

(4)  EM3  -  =  0,  s  =  l,2,.... 

3=0 

Equations  (3)  and  (4)  will  be  called  the  Yule-Walker  equations.  (Many  authors  do  not  include 
(3)  in  “the  Yule- Walker  equations.” ) 

Now  we  shall  show  that  the  roots  of  the  “associated  polynomial  equation  are  less  than  1 
in  absolute  value.  If  p  —  1,  (3)  and  (4)  for  s  =  1  are 


(5) 


cr(0)  +/?x<t(1)  =  c-2, 
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(6) 


<r(l)+/?l<7(0)  =  0. 


These  yield 


CO  (1  -  /3?M0)  =  a2. 

Since  a2  >  0,  it  follows  that  <r(0)  >  0  and  (1-/32)  >  0;  that  is  \0X\  <  1.  (If  a2  =  0  and  «r(0)  >  0, 
then  either  0X  =  1, <r(l)  =  —cr( 0),  and  yt  —  —yt-i  with  probability  1  or  0X  =  — 1,<t(1)  =  cr(0), 
and  j it  =  yt- 1  with  probability  1.) 

Now  suppose  p  =  2  and  the  roots  of 


(8)  x 2  +  0xx  +  /?2  =  0 

are  complex  conjugate  {0{  -  402  <  0);  denote  the  roots  as  ae±iB ,  where  0  <  a  and  0  <  9  <  n. 
Then  (3)  and  (4)  for  s  =  1  and  2  are 


(9)  cr(0)  -  2a  cos  0cr(l)  +  a2cr(2)  =  er2, 

(10)  —2a  cos  9(t(  0)  +  (1  +  a2)<r(l)  =  0, 

(11)  c(2)  —  2a  cos  +  aV(0)  =  0. 
Subtraction  of  (11)  from  (9)  yields 


(12) 


(l-a2)H0)-a(2)]  =  <72. 
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Since  a2  >  0,  we  see  that  a.  <  1  and  <r( 2)  <  <r(0)  because  a (2)  <  o-(O)  by  the  Cauchy-Schwarz 
inequality.  (If  a2  =  0,  <r(0)  >  0,  and  the  roots  are  not  real,  a  =  1.) 

Now  we  consider  p  in  general.  Let  xi,...,xp  be  the  roots  of  the  associated  polynomial 
equation 


(13)  = 

j=0 

Then  the  associated  polynomial  can  be  written 


(14)  Yip*1* 1 = n<*  ~  *«)» 

i— o  i=i 

and  the  stochastic  difference  equation  can  be  written 


p  p 

(is)  -  m)  =  IK1  _  xt£)(yt  -  m), 

3=0  i= 1 

where  £3\ it  =  Vt-j •  If  xi  is  a  real  root,  we  can  write  the  stochastic  difference  equation  as 


(16)  wt  -  xiwt-i  =  ut, 
where 

p 

(17)  wt  =  JJ(1  -  Xi£){yt  -  p). 

4=2 

Then  Ut  is  independent  of  and  {u;{}  is  a  stationary  autoregressive  process  of 

order  1  and  hence  |  a?i  |  <  1.  If  x\  and  x i  are  conjugate  complex  roots,  xi  =  ae'9  and  X2  =  ae~t9 , 
then 
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(18) 


vt  -  2 a  cos  9vt-i  +  a2vt-2  =  ut, 


where 

(19)  =  JJ(1  -  Xi£)(yt  -  fi). 

s=3 

Then  Uf  is  independent  of  vt-i,  vt-2,  ■  •  .,  and  {v*}  is  a  stationary  autoregressive  process  of 
order  2.  From  the  above  treatment  of  the  case  p  =  2,  we  know  that  |xx|  =  |*2|  =  a  <  1.  We 
summarize  the  above  results  in  the  following  theorem. 


Theorem  1.  Let  {yt}  be  a  stationary  stochastic  process  with  mean  £yt  —  p  and  finite 
variance  such  that  (2)  holds  for  suitable  . .  ,,/3p  (/30  =  1)  and  t u?  =  a2  >  0  with  ut  inde¬ 
pendent  of  yt-i,yt-2,  —  Then  ut  is  a  sequence  of  independently  and  identically  distributed 
random  variables.  The  second-order  moments  £ (yt  —  fi)(yt+h  —  p)  —  a{h)  satisfy  (3)  and  (4). 
The  roots  of  the  associated  polynomial  equation  (13)  are  less  than  1  in  absolute  value. 

3.  The  Moving  Average  Representation. 

Now  we  want  to  show  that  yt  can  be  expressed  as  an  infinite  linear  combination  of  the 
independent  U(.  In  the  case  of  p  =  1  we  write 

(20)  yt  =  <j>yt-i  +  ut. 

If  we  replace  yt- 1  in  (20)  by  <j>yt-2  +  ttf-i  (which  is  (20)  with  t  replaced  by  t  —  1),  we  obtain 


(21)  yt  =  ut  + <f>ut-i  + <f>2yt-2- 

If  we  substitute  successively,  we  obtain 
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(22) 


Vt  —  ut  +  +  . . .  +  <j>sut-s  + 


From  (22)  we  obtain 


(23)  £ [yt  —  («{  4-  4>ut- 1  +  . . .  +  <f>‘ tit-a)]2  =  <f>2(*+1)£ 

Since  |^|  =  |/?i|  <  1,  the  right-hand  side  of  (23)  goes  to  0  as  s  increases.  Then  we  write 


OO 

(24)  yt -'^2<j>r ut-r 

r= 0 

and  say  that  the  series  on  the  right  converges  in  the  mean  (of  in  the  mean  square  or  in  quadratic 
mean)  to  yt. 

Now  let  us  turn  to  the  general  case.  The  lag  operator  £  is  a  linear  operator  for  which 
polynomials  are  defined.  The  autoregressive  model  (2)  with  ps  =  0  can  be  written 


(25) 


yt  =  ut. 


Formally  (25)  can  be  inverted  to  give 


=  (gw')u, 


OO 

=  y  \  £sut—3? 

3=0 


where  the  coefficients  Sq,Si,...  are  defined  by  the  identity 
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(27) 


1  = 

i=0  4=0 

P  oo 

=  T,T,Pis*zj+3 

J  = 0  4=0 
oo  min(r,p) 

=  E  E 

r= 0  j'=0 

where  r  =  y  +  s  and  the  limits  of  the  last  sum  on  j  depend  on  the  value  of  r.  The  coefficient 
of  each  power  of  z  on  the  left-hand  side  of  (27)  is  equal  to  the  coefficient  of  that  power  on  the 
right-hand  side.  Thus 

1  =  Pq6q  =  So, 

(28)  0  =  /Mi  +  /Mo  =  Si  +  f3\, 

0  =  /Vp-i  +  PiSp- 2  +  •  ■  •  +  fip-iSo, 

(29)  0  =  floSt  +  /Mt-i  +  . . .  +  /3p6t-p,  t  =  p,p  +  1, - 

We  look  for  a  sequence  So,  Si,...  that  satisfies  the  p  equations  (28)  and  the  set  of  equations 

(29) .  Equations  (28)  determine  So, . .  .,Sp-i  successively  and  (29)  determines  Sp,Sp+ in 
turn;  the  solution  to  these  equations  exists  and  is  unique. 

Does  (26)  converge  in  the  mean?  We  note  that 

OO 

(30)  (I-*,-*)"1  ==£*?*• 

4  =  0 

converges  uniformly  for  \z\  <  ^/\x,\  for  0  <  7  <  1.  Hence, 
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oo  p 

(si)  ]r  =  (i +&Z+...+ vt1  =  n 

3=0  i=l 

converges  uniformly  for  \z\  <  7/  max^i.p  |i,|,  which  is  greater  than  1  if  7  is  large  enough. 
Hence,  Ss  —*  0  as  s  — *  00.  We  write  for  m  >  p 


m  m  p 

(32)  =  i 

3=0  3=0  j=0 

m  p 

=  PjfisVt-a-j 

s=Q  j= 0 
m+p  min(r,p) 

=  Pjfir-jVt-r 

r=0  j'=max(0, r—m) 
m+p  p 

=  Vi  "h  PjK-jVt-r 

r=?7i-|-l  j=r—m 

by  comparison  with  (27).  From  this  we  obtain  for  m  >  p 


(m  \  2  (  m+p 

yt~YlSsUt-s)  Y1  PiSr-jVt-r 

3=0  /  yr=m+l  j 

The  right-hand  side  is  a  quadratic  form  in  6m+i-p, . . . ,  Sm+p.  Since  Ss  — *•  0  as  s  — >  00,  the 
right-hand  side  converges  to  0  and  (26)  converges  in  the  mean  as  m  — ►  00. 


Theorem  2.  If  yt,  t  =  ..  .,—1,0,1,...,  satisfies  (2)  with  ut  independent  of  yt-i,yt-2,  ■ . . 
and  £yt  =  (i,  then 

00 

(34)  yt  =  p  +  ^2  Ssuts 

3=0 

converges  in  the  mean,  where  5q  =  l,5i, . . .  are  determined  by  (28)  and  (29). 

The  expression  (34)  is  called  the  moving  average  representation  of  the  autoregressive 
process  {ye}. 
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We  shall  now  find  an  explicit  expression  for  Ss  when  the  roots  of  the  associated  polynomial 
equation  are  distinct. 

The  equation  (29)  is  a  homogeneous  difference  equation.  (It  is  homogeneous  of  degree 
1  in  the  St’s,  and  it  is  a  difference  equation  because  it  can  be  written  as  a  polynomial  in 
V  =  1  —  £  operating  on  St.)  The  case  p  =  1  suggests  that  a  solution  to  the  homogeneous 
difference  equation  (29)  may  be  of  the  form  St  =  x*.  Then  the  right-hand  side  of 


(35)  =  xt-rJ2/3jx*’-i 

3=0  j=0 

is  0  if  i  satisfies  the  associated  polynomial  equation  (13);  that  is,  if  x  is  one  of  the  roots 
xi, . .  ,,xp  of  (13).  It  follows  that 

p 

(36)  St  =  ^  kill,  t  =  0,1,..., 

i=l 

is  also  a  solution.  If  we  substitute  (36)  into  (28),  we  obtain 


1  ~~ 

i=l 

(3?)  0 = E  k*x< + &  E ki  =  X>«- + &)*•-> 

«=i  i=i  i=i 


q  =  E  kix% 1 + X!  kiXi  2  +  •  •  ■ + Pp-i  X ki 
1  =  1  1=1  8=1 

The  detrminant  of  the  coefficients  of  ki, . . . ,  kp  in  (37)  is 
(38) 


1  1  ...  1 

Xl  +  Pi  X2+  01  ...  Xp+  0 1 


Xl  1  +  0lZl  2  +  ...  +  0p- 1  X%  1  +  01^2  2  +  ---  +  0p-l  •••  1  +  01%P  2  +  ...  +  0p- 1 


— X(*r 1 + 0ii?i 2 +.. .+ 0p-i)ki. 
8  =  1 
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1  0  ...  0 

1  1  ...  1 

Pi  1  ...  0 

0 

x2  •  ■  •  xp 

Pp—1  ftp— 2  •  •  •  1 

v—1  »—  1 

x X  c  .  0  Xp 

= m*  -  xi)- 

3<i 


This  (Vandermonde)  determinant  is  different  from  0  if  and  only  if  the  roots  are  distinct.  Hence, 
there  is  a  unique  solution  to  (28)  of  the  form  (36)  if  and  only  if  the  roots  of  (13)  are  distinct. 
Then  8t  =  ]C<=i  is  a  solution  to  (28)  and  (29)  and  is  the  only  solution  to  (28).  Since  8t  is 
uniquely  determined  by  5t_i, . . . ,  8t-p  in  (29),  t  =  p,p+  1, . . .,  the  solution  of  8t  =  k,x\  is 
the  unique  solution  to  (28)  and  (29)  if  the  roots  of  (13)  are  distinct. 


Theorem  3.  The  unique  solution  to  the  homogeneous  difference  equation  (29)  satisfying 
the  conditions  (28)  is  8t  =  kx*,  where  ki,...,kp  satisfy  (37)  and  x\,...,xp  are  the  roots 
of  (13),  if  xi, . . . ,  xp  are  distinct. 

If  p  =  1,  8r  =  (— /?i)r,  an  exponential  function  of  r.  If  p  =  2  and  x\  and  x2  are  different, 
h  =  x\/{xi  ~  x2),k2  =  ~x2/{xi  -  x2),  and 


(39) 


,r+ 1 


—  X. 


r+1 


xi  -  x2 


r  —  0,1,. 


If  Xi  and  x2  are  real,  8r  is  a  linear  combination  of  two  exponential  functions  of  r.  If  the  roots 
are  complex,  we  may  write  them  x\  —  ae'8  and  x2  =  ae~'e .  Then 


(40) 


et0(r+l)  _  eiS(r+l) 


rsin  9{r  +  1) 

a  - r — - — - 

sin  6 


a  damped  sine  function  of  r. 
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4.  The  Autocovariance  Sequence. 


Now  we  find  an  explicit  expression  for  {<r(h)}  when  the  roots  of  the  associated  polynomial 
equation  are  distinct.  To  do  this  we  first  demonstrate  that  (3)  and  the  first  p  equations  in 
(4)  determine  <r(0), o-(l),  ...,a(p)  uniquely  regardless  of  whether  the  roots  are  distinct.  Since 
<r(h)  =  a(—h),  h  =  1, . . .  ,p  —  1,  these  p  +  1  equations  can  be  written 


(41)  a(0)  +  0ia(l)  +  . . .  + /3p<r(p)  =  a2, 

/M  o)  +  (i  +  AMi)  + . . .  +  0p<t(p  - 1)  =  o, 


/3p<r(0)  +  /?p_i<r(l)  +  . . .  +  <r{p)  =  0. 
The  determinant  of  the  matrix  of  coefficients  is 


1 

Pi 

Pi 

.  .  .  Pp- 1 

Pp 

Pi 

1  +  ^2 

Pz 

Pp 

0 

Pi 

Pi  +  Pz 

1  4"  Pi 

0 

0 

Pp-l 

Pp- 2  d"  Pp 

Pp-Z 

1 

0 

Pp 

Pp-l 

Pp- 2 

•  Pi 

1 

If  we  multiply  the  (j  +  l)st  column  by  x\  +  x1 ;  and  add  the  result  to  2  times  the  first  column, 
j  —  1, ...  ,p,  we  obtain  as  the  first  column  the  transpose  of 


(43)  (  faxi 3  ’  xi 1  +  xij2fcxi:’---’xiPYl  Pix  l  +  J 

\j=0  j= 0  j= 0  j— 0  j— 0  j= 0  / 

p 

3=0 
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because  2y=o  /3jXx  3  =  0.  We  assume  that  /3P  7^  0  and  hence  xy  7^  0,  j  =  1, . . .  ,p.  Thus 

(44)  &*£  =  IB1  “  *«*i) 

i=o  *=i 

is  a  factor  of  the  determinant.  Since  this  operation  can  be  done  with  any  root, 

(45)  na- ****) 

i<3 

is  a  factor.  The  term  in  the  determinant  of  highest  degree  in  xj, . . . ,  xp  is 

(46)  ±^+i=±n<+i> 

«=i 

which  is  the  degree  of  (45).  Since  the  determinant  is  1  for  xi  =  . . .  =  xp  =  0,  the  determinant 
is  (45).  This  is  different  from  0  because  |x,-|  <  1,  i  =  1, . .  .  ,p.  Thus  the  Yule- Walker  equations 
determine  <r(0), . . . ,  a(p)  uniquely  as  the  solution  to  (41). 

The  equation  (4)  is  a  homogeneous  difference  equation  for  <r(h,),  h  —  —  (p—  1),  —  (p—2), . . ., 
and  <x(/i)  =  1  °ixi  satisfies  it  for  any  ci, . . . ,  cp.  However,  if  the  roots  are  distinct  there  is 

only  one  set  of  ci, . . . ,  cp  that  satisfies  a(h)  =  cixi  for  h  =  —  (p  —  1), . . . ,  0;  that  is, 


1 

§ 

'  1 

1 

1  ' 

'Cl' 

(47) 

a(-l) 

= 

xil 

x-1 

xp 

C2 

,<r(-p+  1). 

-Cp-1) 
x 2 

-(p-1) 

xp 

.cp. 

The  determinant  of  the  matrix  of  coefficients  of  cj, . . .,  cp  in  (47)  in  the  Vandermonde  deter¬ 
minant  rW*,-1  —  z/1).  which  is  nonzero. 


Theorem  4.  If  the  roots  of  the  associated  polynomial  equation  are  distinct  and  (3P  7^  0, 
the  autocovariances  are  given  by  <r(/i)  —  Y%=t  °ixi >  h  =  1  —  p,  2  —  p, . . .,  where  cj, . . . ,  cp  satisfy 
(47)  or  equivalently  (3)  and  cr(/i)  =  a(—h),  h  =  1, . .  .,p  —  1. 

If  p  =  1,  cr(h)  =  (— /?i)A(72/(1  -  /?2),  h  —  0, 1, . . .,  an  exponential  function  of  h.  If  p  =  2 
and  xi  and  x<i  are  different, 
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(48) 


m  _  (  *?+1  _ 

( Xi  -  £2)(l  -  X1X2)  1  -  x\  1  -  x\  J  ' 

If  x\  and  £2  are  real,  <r(h)  is  a  linear  combination  of  two  exponential  functions.  If  p  =  2  and 
£1  and  £2  are  conjugate  complex,  ae±t9,  then  (48)  can  be  written 


(49) 


_  a2ah  [sin  9{h  +  1)  -  a?  sin  9{h  —  1)] 
(1  —  a2)  sin  6  [l  —  2a2  cos  29  +  a4] 

_  tr2ahh  cos(6h  —  tp) 

(1  —  a2)  sin  9\Jl  —  2a 2  cos  29  +  c*4  ’ 


where 


(50) 


tani f)  = 


(1  —  a2)  cosO 
(1  4-  a2)  sin  9 


Here  we  have  assumed  that  {yt}  is  strictly  stationary  and  that  ut  defined  by  (2)  is  in- 
depenent  of  yt_1,t/t_2, —  Then  the  u<’ s  are  independently  identically  distributed.  A  similar 
development  can  be  carried  out  on  the  assumptions  that  {yt}  is  stationary  in  the  wide  sense 
and  that  ut  is  uncorrelated  with  yt_i,yt_2, —  Then  the  ut’s  are  uncorrelated  and  have  the 
same  variance. 

An  alternative  approach  to  the  stationary  autoregressive  model  is  to  assume  {ttf}  consists 
of  independently  identically  distributed  random  variables  with  £ut  =  0  and  £u2  =  a2 .  Then 
define  yt  by  (34),  where  {<Ss}  satisfies  (28)  and  (29)  for  some  /?i, . .  .,/3p  with  the  roots  of  (13) 
less  than  1  in  absolute  value.  Then  {yt}  satisfies  (2)  and  ut  is  independent  of  yt-i,  yt-2,  •  ■  •• 


5.  The  Direction  of  Time. 


What  gives  time  the  direction  is  that  Uf  is  independent  of  the  observable  past: 
yt_i,  yt_2, . . ..  If  {yt}  is  Gaussian  (that  is,  all  finite  distributions  are  normal),  then  {yt}  is 
determined  by  £yt  —  n  and  £{yt  —  fi)(yt+h  -  m)  =  <r{h),h  =  0,1,....  Since  u(h)  =  a(-h), 
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the  covariance  function  does  not  distinguish  between  the  sequence  t  =  . . .,  —  1, 0, 1, . . .  and  the 
sequence  t  =  . . . ,  1, 0,  —  1, . . ..  In  fact,  for  and  cr2  given,  the  random  variable 

p 

(51)  'ZZPjyt+k  =  wt 

3=0 

is  independent  of  y*+i,  yt+z,  ■  ■ . ,  {tut}  is  a  sequence  of  independently  identically  distributed  ran¬ 
dom  variables,  and  the  Yule-Walker  equations  hold.  For  example,  if  <r(h)  =  (— h  = 
±1,  ±2, . . . ,  then  (51)  is 

(52)  yt  +  Piyt+i  =  u>t, 
and  yt  =  Y,7=o{-PiYwt+s- 

Let  us  look  briefly  at  the  case  in  which  some  roots  of  the  polynomial  equation  are  larger 
than  1  in  absolute  value.  Suppose  |ii|  >1,  *  =  1, . . . ,  q,  |z,|  <  1,  i  =  q  +  1, . . .  ,p.  We  write 
the  stochastic  difference  equation  as 

p 

(53)  ut  =  Y^Pr£ryt 

r=0 

P 

=  IL'1  -  x<£)yt‘ 

The  inverse  of  (53)  is 


(54) 


yt  =  ]J(i  -  xiZ)  1ut 

i=l 
9 


=  n(i-^-cr‘  n  (i-^r1 


«=1 

9 


<=9  +  1 


*=1  x  %/  v  *  7  <=0+1 


where  P  =  £-1.  Each  term  (1  —  x^1P)~1  can  be  expanded  in  a  power  series  in  P ,  and  each  term 
(1  —  XiZ)-1  can  be  expanded  in  a  power  series  in  Z-  If  1  <  q  <  p,  the  operator  will  be  a  power 
series  in  Z  (=  P-1)  and  in  P  (=  £_1);  the  indicated  infinite  series  in  . . . ,  ut_i,  ut,  ut+x, . . . 
will  be  doubly  infinite.  If  q  =  p  (all  roots  in  absolute  value  greater  than  1),  the  operator  is 
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an  infinite  power  series  in  P  (only  nonnegative  powers);  the  indicated  infinite  series  in  the 
random  variables  involves  U{+P,«t+P+ i, —  This  discussion  is  in  purely  formal  terms,  but  it 
can  be  justified  in  the  manner  used  for  the  case  of  all  roots  less  than  1  in  absolute  value.  In 
fact,  if  the  linear  form  n£_j.(l  —  xi£)vt  is  replaced  by  the  linear  form  (for  any  x,-’s) 

(55)  n<^  -  n  (i  -  xx)v, = «; 

t=l  i=r+l 

the  residuals  ujf  are  also  uncorrelated  (though  not  necessarily  independent). 

Now  consider  the  special  case  of  one  root  equal  to  1 

(56)  yt-Vt-i  +  ut, 

and  assume  £ut  =  0,  £u2  =  a2.  Then 

(57)  yt  ~  yt-s  =  ut  +  «t-i  +  •  •  •  +  tif-,,+1, 
and 


(58)  £  (yt  -  yt-3)2  =  £  yt  +  £  yt-s  -  2£  ytyt-s  =  sa 2. 

If  the  process  is  stationary,  £yt2  =  £yt2_s;  from  (58)  we  obtain 


(59)  £ytyt-3  =  Syt  -  ^s<72,  s  =  i,2,.... 

This  can  hold  for  all  s  >  0  only  if  a2  =  0  and  then  yt  =  yt-s  with  probability  1.  Intuitively, 
we  see  that  unless  the  variance  of  ut  is  0,  (56)  implies  that  the  variance  of  yt  increases  with  t, 
but  this  fact  is  contrary  to  stationarity. 

A  more  general  case  is 

(60)  (l-£)n(l-s,-£)yt  =  ut, 

*=2 
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where  \xi\  ^  1, *  =  2, .  .  .,p.  If  11^=2 —  xi£)yt  =  zti  then  (1  —  t)z%  =  u*  and  zt  —  zt~s  with 
probability  1,  say  zt  =  z.  Thus 

p 

(61)  J][(l  -  XiL'jyt  =  z, 

i=  2 

and  yt  =  Y2^L-<x>  that  is,  yt  =  yt-3  with  probability  1.  (z  can  be  a  random  variable.) 

6.  Fluctuations  of  the  Time  Series. 

The  typical  time  series  generated  by  an  autoregression  model  fluctuates  up  and  down.  Its 
oscillations  are  not  regular,  but  tend  to  have  an  average  length  which  depends  on  the  difference 
equation.  If  we  think  of  (2)  as  generating  the  series  for  successive  values  of  t  we  see  that  each 
set  of  p  yt’s  directly  affects  the  next  yt.  If  p  =  2,  we  can  write  the  equation 


(62)  yt  =  <f>iyt-i  +  fayt-i  +  «t 

=  {4>i  +  4>2)yt-i  —  —  yt-?.)  +  v.t, 

which  indicates  that  the  direct  effect  of  preceding  yr’s  on  yt  involves  the  value  of  yt-i  and 
the  change  between  t/t_2  and  yt-i-  This  effect  will  generally  have  a  tendency  to  produce 
fluctuations. 

Another  way  of  looking  at  the  process  is  in  terms  of  the  representation  of  yt  =  ^r^t-r- 

A  given  u3  will  affect  a  subsequent  ys+q  according  to  the  coefficient  Sq.  Since  these  coefficients 
oscillate,  the  effect  on  successive  yr’s  fluctuates,  tending  to  produce  fluctuations  in  the  series 
Vt- 

The  fluctuating  behavior  is  also  indicated  by  the  covariance  function.  Since  cr(s)  ±  0 
(usually) ,  there  is  some  statistical  association  between  yt  and  yt+s ,  though  it  tends  to  decrease 
as  s  (>  0)  increases.  This  association  (measured  by  the  correlation  coefficient)  tends  to  fluc¬ 
tuate.  For  example,  if  there  is  a  pair  of  complex  roots,  the  trigonometric  functions  oscillate 
and  the  association  may  increase  with  s  in  some  intervals  (of  s). 
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7.  Partial  Autocorrelations., 


A  stationary  stochastic  process  {t/t}  with  finite  second  moment  defines  an  autocovari¬ 
ance  sequence  {<r(/i)}.  In  turn,  each  set  <r(0),  o-(l), . . . ,  a(p)  defines  a  set  of  coefficients 
<t>\ (p), • •  • ,  4>p(p)  by  the  p  Yule- Walker  equations 

(63)  ~j)ii(p)  =  CT(S)>  s  =  l,...,p. 

i=i 

We  shall  now  show  that  the  autoregressive  coefficient  <j>p(p)  in  the  fitted  AR(p)  model 
is  identical  to  the  partial  correlation  coefficient  between  yt  and  yt-p  “holding  yt- 1, . .  Vt~P+i 
fixed.”  This  partial  correlation  coefficient  is  the  correlation  between  the  residual  of  yt  regressed 
on  2/t-l,  •  •  • ,  Pt-p+i  and  the  residual  of  yt-p  regresssed  on  yt-i, . . . ,  yf_p+1 .  We  suppose  £yt  =0. 
Define 


(64) 

„(P-1)  _ 
~{_1  _ 

yt-i 

yt-2 

,  SytVtP  1]  =  <v- i  = 

o-l1) 

<t(2) 

.yt-p+i . 

Mp~  i). 

<t{0)  cr(l) 

•  ^(p-2)] 

(65) 

fV (p>—1)r,(P— 1)'  _  -c-.  _ 

Cyt-1  yt- 1  —  £jp-\  — 

r**  r*j  (nj 

<t(1)  a(0) 

c>r 

i  ... 

3 

V 

-a{p  ~  2)  cr{p-  3)  . 

g 

Then  the  vector  of  regression  coefficients  <j>p-i  =  -  1), ,  <f>p-i(p  -  1)]*  of  yt  on  yj^1)  = 

{yt- 1, .  •  • ,  is  the  solution  to 

(66)  Sp-i<£p-i  =  <Tp-i- 

The  vector  of  regression  coefficients  of  yt-p  on  yt-p+ 1, . . . ,  yt-i  is  defined  by  the  same  equation. 
(The  covariances  of  yr  and  y3  depends  only  on  |r  -  s|,  not  the  sign  of  r  -  s.)  Thus  the  vector  of 
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regression  coefficients  of  y*_p  on  yj^.^  is  4>p-i  =  [4>p-i(p  —  1), . . . ,  —  1)]*.  The  variances 

of  the  residuals  are 


(67) 

£{yt  -  ^p-iyi-i15)2  =  <y(o)  =  < 

/V  ^  > 

K-i^p-i^p-u 

»v  ^ 

(68) 

t (yt-p  -  ^p-itfc15)2  =  (o)  -  < 

bp— ].SP—  l&p— 1  j 

these  two  are  the  same.  The  covariance  between  the  two  residuals  is 

(69)  £{yt  -  -  4>p-iy[p~i1))  =  *( p )  -  $,_iEP-i^P-i. 

/v/  ru  ^  /s j  ~ 

The  partial  autocorrelation  between  yt  and  yt_p  given  y^1^  is 


(70) 


PACF(p)  = 


<r(p)  - 

rvy  ^  ^ 

<t{ ti)  -  4>p-\2p-i<!>p-i 


Now  let  us  write  the  equation  for  the  vector  of  regression  coefficients  of  yt  on  yt-i, 
in  partitioned  form 


i  Vt—% 


(71) 

where  &p-i 


Sp-1  ®p— i 

>W(p)' 

,£P-i  CT(°). 

.  Mp )  . 

.  <Kp) . 

a(p  —  1) , .  „ .  s  cr(l)]/.  The  partitioned  equations  are 


(72) 


SP-1<£(  ^(p)  +  ^P-l^*P(p)  —  Vp—h 

~  r*j  ~  ~ 


(73)  ^P-i<£(1)(p)  +  *(0)^P(p)  =  <r(p). 

The  first  equation  yields 
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(74) 


f1](p)  =  S^Vp-l  -  s;Vp-i<£p(p) 

=  <f>p- 1  -  <t>p{p)<f>p-l- 


Substitution  into  (73)  yields 


(75) 


<r(0)  -  ^p-iSpi^p-i 


Mp)  =  ct(p)  ~  2P-iffP-i- 


p-i' 


Comparison  of  (70)  and  (75)  shows  that  <t>p{jp)  —  PACF(p). 

These  equations  lead  to  the  Levinson-Durbin  recursive  relations  (74)  and 


(76) 


Mp)  = 


*00  -  z'p-iMi 
<r{0)  -  cr'p_iMl 


The  parameters  cr(0)  and  {<f>v{p)}  are  equivalent  to  the  sequence  {(r(h)}. 


8.  Prediction. 


The  autoregressive  process  (2)  for  p  =  0  can  be  written 
(77)  yt  =  <f>iyt-i  +  -  -  •  +  4>Pyt-p  +  ut, 

where  ut  is  independent  of  yt-i,yt-2, —  Then  the  conditional  expectation  of  yt  given 

yt-i,yt-2 •  is 


(78) 


£  {yt\yt-i,  yt- 2,  •  •  •}  —  <t>iyt-i  +  •  •  -  +  <t>Pyt-P- 


This  conditional  expectation  is  the  best  predictor  of  yt  based  on  the  past.  Its  variance  is 


(79) 


£ +  •  •  •  +  4>Pyt-p  ~  yt]  =  £u;  =  a2. 
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The  mean  squared  error  of  any  other  predictor,  say  Vt-2>  •  ■  •)>  is 

(80)  £  [f(vt-u  vt-2,  ■  ■  ■)  -  yt}2 

=  £[f(yt-i,yt-2,  ■■■)-  <t>iyt-i  -  ...  -  <f>Pyt-P  -  ut}2 
=  £u\  +  S [/(yt-i,yt-2,.-.)  -  4>iyt-i  -  ■■■-  4>Pyt-P\2- 

Theorem  5.  The  predictor  of  yt  based  on  yt-i ,  Vt-2,  ■  -  •  with  minimum  mean  square 
error  for  the  stationary  stochastic  process  satisfying  (2)  for  which  the  roots  are  less  than  1  in 
absolute  value  is  (58). 
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